Field of the Invention
The present invention relates to digital image processing and particularly to an image processing technique for performing image processing on a photographed image.
Description of the Related Art
A known photographic technique among various techniques of photography using a camera involves emphasizing a subject of interest by intentionally blurring portions other than the subject (such as a background). However, when performance of an optical system is not particularly high as in the case of a compact camera, a depth of field cannot be reduced and a sufficient blur cannot be created. In consideration thereof, techniques for generating an image with a blurred background or the like by subsequently adding a blur by digital image processing are recently being proposed. For example, an image pickup apparatus according to Japanese Patent Application Laid-open No. 2011-091570 determines a position on an image of a subject that is an in-focus object as a reference point and applies a blurring process so that the greater the distance on the image from the reference point, the greater the blur. In addition, Japanese Patent Application Laid-open No. 2000-207549 discloses an image processing apparatus that generates an image on which an appropriate blurring effect is applied based on data of a three-dimensional image photographed using a three-dimensional measurement camera and on virtual camera parameters and a focal position specified by a user.
In the method disclosed in Japanese Patent Application Laid-open No. 2011-091570, a size of a blur is determined in accordance with a position on an image instead of a depth distance. Therefore, there is a possibility of generating an unnatural image in which, for example, a blur is added to an object even if a depth distance of the object is the same as that of an in-focus object (in other words, even if the object exists at a focal position). On the other hand, although the method disclosed in Japanese Patent Application Laid-open No. 2000-207549 is conceivably capable of producing a high-quality blurred image, the method requires the use of a three-dimensional measurement camera that is a special apparatus. Therefore, it is difficult to apply the method to general small-sized cameras and the like. Another problem is that conditions such as camera parameters and a focal position must be specified by the user which makes handling difficult.
In consideration thereof, the present inventors have considered photographing an image by generating data that records a depth distance of each point in the image (this data will be referred to as a depth map or a distance map) and using the depth map in a blur adding process. Once a focal position (an in-focus distance) of a photographed image is known from autofocus information, a deviation (defocus amount) from a focal position at each point in the image can be calculated by calculating a difference between the focal position (the in-focus distance) and distance information of each point on the depth map. By controlling a size of a blur based on the defocus amount, an image in which blurring increases as a deviation from a focal position increases can be generated while maintaining sharpness of an in-focus object at the focal position.
However, in furthering the consideration, the present inventors discovered that the method described above has the following problems. Although a depth map can be created by, for example, various methods including a depth from defocus (DFD) method and a stereo method, in either case, an error is created in a distance estimation result due to the influence of noise and spatial frequency. A detailed description will now be given with reference to FIG. 11. It is assumed that a sphere 80, a trigonal pyramid 81, and a column 82 are respectively placed at positions of 3 m, 4 m, and 10 m from a camera, and photography is performed and a depth map is created by bringing the sphere 80 into focus. Reference numeral 83 denotes a photographed image and reference numeral 84 denotes an example of a depth map. While the depth map 84 contains records of measurement results with respect to 24 (4×6) areas, a certain amount of error is created as compared to actual object distances. Reference numeral 85 denotes defocus amounts of the respective areas calculated based on the depth map 84 and a focal position (3 m) obtained from autofocus information. Due to the errors in the depth map, a defocus amount of an area (a dotted portion) of the sphere 80 that is an in-focus object has exceeded a defocus amount of an area (a hatched portion) of the trigonal pyramid 81 that is in the background. Adding a blur in accordance with such defocus amounts causes the sphere 80 that is a subject brought into focus by a photographer to blur and generates a blurred image 86 that is contrary to the intention of the photographer.
In addition to a blur adding process, similar problems arise with respect to, for example, a trimming process of a subject. In other words, when the intention is to trim (crop) the area (the dotted portion) of the sphere 80 that is an in-focus object using the depth map 84 and the focal position obtained from autofocus information, the area (the hatched portion) of the trigonal pyramid 81 with a small defocus amount ends up being trimmed. As a result, a subject intended by the photographer cannot be trimmed.